Validity Threats in Empirical Software Engineering Research - An Initial Survey

In judging the quality of a research study it is very important to consider threats to the validity of the study and the results. This is particularly important for empirical research where there is often a multitude of possible threats. With a growing focus on empirical research methods in software engineering it is important that there is a consensus in the community on this importance, that validity analysis is done by every researcher and that there is common terminology and support on how to do and report it. Even though there are previous relevant results they have primarily focused on quantitative research methods and in particular experiments. Here we look at the existing advice and guidelines and then perform a review of 43 papers published in the ESEM conference in 2009 and analyse the validity analysis they include and which threats and strategies for overcoming them that were given by the authors. Based on this analysis we then discuss what is working well and less well in validity analysis of empirical software engineering research and present recommendations on how to better support validity analysis in the future. Keywords-Validity threats, Empirical research, Software engineering, Research methodology